Detect Medicare Fraud in the US through Classification Model¶
Medical fraud is a major contributor to inflating expenses and waste in the US healthcare system. Forbes identifies the industry as among the most vulnerable to fraud & cyber-attacks. This investigation explores physician-level Medicare and Medicaid data through exploratory data analysis and a classfication model, hoping to show how big data can be used to prevent this problem.
The inspiration for this project came from my coleague at IBM, whose mother was a victim of medical ID theft. Although the problem of ID theft is slightly different (would require transctional data), CMS and LEIE data are among the best publicly available sources for healthcare and thus were used as a proof-of-concept. It reaffirms that, similar to credit card fraud detection, fraudulent practices in healthcare can be prevented and alleviated. The most vulnerable demographics, dependent elders of 65 years and older, would benefit the most from this technology.
Healthcare fraud in the news:
- 24 Charged in $1.2 Billion Medicare Scheme: https://www.nytimes.com/2019/04/09/us/billion-dollar-medicare-scam.html
- Medical Identity Theft: https://www.consumerreports.org/medical-identity-theft/medical-identity-theft/
Data sources:
- CMS Data (2012-2013, 2015-2016): https://www.cms.gov/newsroom/data
- LEIE Label Data (monthly 2018-2019): https://oig.hhs.gov/exclusions/exclusions_list.asp
In cleaning and combining the two datasets, I followed closely previous literature, which are both listed in the "References" section.
Menu¶
In [1]:
import os
import pandas as pd
In [2]:
# Set up data directory
CWD = os.getcwd()
cms_data_dir = os.path.join(CWD, 'CMSData')
In [3]:
# Some years columns are capitalized and other years the columns are lowercase:
capitalization_dict = {
'2012': str.upper,
'2013': str.upper,
'2014': str.lower,
'2015': str.lower,
'2016': str.upper,
'2017': str.lower,
}
1. CMS Part B dataset¶
In [4]:
# Set dtypes based on https://www.cms.gov/Research-Statistics-Data-and-Systems/Statistics-Trends-and-Reports/...
#Medicare-Provider-Charge-Data/Physician-and-Other-Supplier2017
partB_dtypes = {
'npi': 'str',
'nppes_provider_last_org_name': 'str',
'nppes_provider_first_name': 'str',
'nppes_provider_mi': 'str',
'nppes_credentials': 'str',
'nppes_provider_gender': 'str',
'nppes_entity_code': 'str',
'nppes_provider_street1': 'str',
'nppes_provider_street2': 'str',
'nppes_provider_city': 'str',
'nppes_provider_zip': 'str',
'nppes_provider_state': 'str',
'nppes_provider_country': 'str',
'provider_type': 'str',
'medicare_participation_indicator': 'str',
'place_of_service': 'str',
'hcpcs_code': 'str',
'hcpcs_description': 'str',
'hcpcs_drug_indicator': 'str',
'line_srvc_cnt': 'float64',
'bene_unique_cnt': 'float64',
'bene_day_srvc_cnt': 'float64',
'average_medicare_allowed_amt': 'float64',
'average_submitted_chrg_amt': 'float64',
'average_medicare_payment_amt': 'float64',
'average_medicare_standard_amt': 'float64',
}
In [5]:
# Get dfs for all years - TAKE A FEW MINUTES
years = ['2012','2013','2015','2016']
dfs = []
for year in years:
file = os.path.join(cms_data_dir, f'cms{year}.txt')
dtypes = dict(zip(list(map(capitalization_dict[year], partB_dtypes.keys())), list(partB_dtypes.values()))) #get correct column capitalization and dtype
df = pd.read_csv(file, delimiter='\t', dtype=dtypes)
df.columns = map(str.lower, df.columns) # make all variable names lowercase
df['year'] = year #add Year column
dfs.append(df)
In [6]:
# Concatenate
partB_df = pd.concat(dfs, axis=0, ignore_index=True, sort=False)
partB_df.shape
Out[6]:
In [7]:
# Remove rows corresponding to drugs because LINE_SRVC_CNT for them is not a desirable count
partB_df = partB_df[(partB_df['hcpcs_drug_indicator'] == 'N')]
partB_df.shape
Out[7]:
In [8]:
# Drop missing NPI and HCPCS - "Medicare fraud detection using neural networks" (Johnson, Khoshgoftaar 2019)
# This means dropping 2014 and 2016 - both did not have HCPCS Code
partB_df = partB_df.dropna(subset = ['npi','hcpcs_code'])
partB_df.shape
Out[8]:
In [9]:
# Keep variables based on "Medicare fraud detection using neural networks" (Johnson, Khoshgoftaar 2019)
partB_variables_to_keep = [
'npi',
'provider_type',
'nppes_provider_city', # keep
'nppes_provider_zip', # keep
'nppes_provider_state', # keep
'nppes_provider_country', # keep
'hcpcs_code', # not in paper but kept
'hcpcs_description', # not in paper but kept
'hcpcs_drug_indicator', # not in paper but kept
'place_of_service', # not in paper but kept
'nppes_provider_gender',
'line_srvc_cnt',
'bene_unique_cnt',
'bene_day_srvc_cnt',
'average_submitted_chrg_amt',
'average_medicare_payment_amt',
'year' # need Year for labeling
]
partB_df = partB_df[partB_variables_to_keep]
In [10]:
partB_df.head()
Out[10]:
In [11]:
partB_df.loc[partB_df['npi'] == '1003000142'][['npi',
'provider_type',
'place_of_service',
'line_srvc_cnt',
'average_submitted_chrg_amt',
'year']][:5]
Out[11]:
In [12]:
partB_df['year'].value_counts()
Out[12]:
In [13]:
# Write all combined CMS to csv
#partB_df.to_csv('combined-partB-data-v2')
2. LEIE Dataset¶
In [14]:
leie_data_dir = os.path.join(CWD, 'LEIEData')
In [15]:
leie_dtypes = {
'LASTNAME': 'str',
'FIRSTNAME': 'str',
'MIDNAME': 'str',
'BUSNAME' : 'str',
'GENERAL': 'str',
'SPECIALTY': 'str',
'UPIN': 'str',
'NPI': 'str',
'DOB': 'str',
'ADDRESS': 'str',
'CITY': 'str',
'STATE': 'str',
'ZIP': 'str',
'EXCLTYPE': 'str',
'EXCLDATE': 'int64',
'REINDATE': 'int64',
'WAIVERDATE': 'int64',
'WVRSTATE': 'str',
}
In [16]:
#LEIE data is monthly between 01/2018 (1801) - 12/2019 (1912)
year_months = ['1801','1802','1803','1804','1805','1806','1807','1808','1809','1810','1811','1812',
'1901','1902','1903','1904','1905','1906','1907','1908','1909','1910','1911','1912']
dfs = []
for year_month in year_months:
file = os.path.join(leie_data_dir, f'leie{year_month}-excl.csv')
df = pd.read_csv(file, dtype=leie_dtypes)
df.columns = map(str.lower, df.columns)
dfs.append(df)
In [17]:
# Concatenate
leie_df = pd.concat(dfs, axis=0, ignore_index=True, sort=False)
leie_df.shape
Out[17]:
In [18]:
leie_df.head()
Out[18]:
In [19]:
# Drop NPI = 0, which means missing - A LOT ARE MISSING, which is a problem for the data
leie_df = leie_df[leie_df['npi'] != 0]
leie_df.shape
Out[19]:
In [20]:
# Keep exclusions most related to Fraud
exclusions_to_keep = [
'1128a1',
'1128a2',
'1128a3',
'1128b4',
'1128b7',
'1128c3Gi',
'1128c3gii',
]
leie_df = leie_df[leie_df['excltype'].isin(exclusions_to_keep)]
leie_df.shape
Out[20]:
In [21]:
leie_df['excltype'].value_counts()
Out[21]:
Type definitions of physician fraud:
- 1128a1: Conviction of program-related crimes
- 1128a2: Conviction of relating to patient abuse or neglect
- 1128a3: Felony conviction relating to healthcare fraud
- 1128a4: License revocation, suspension, or surrender
- 1128a3: Fraud, kickbacks, other prohibited activities
In [22]:
# Write all combined LEIE to csv
#partB_df.to_csv('combined-leie-data')
3. Combine/Label Data¶
In [23]:
from datetime import datetime, timedelta
import numpy as np
In [24]:
# Convert to datetime
leie_df['excldate'] = pd.to_datetime(leie_df['excldate'], format='%Y%m%d', errors ='ignore')
In [25]:
# Round excl date to the nearest year Johnson & Khoshgoftaar (2019)
def round_to_year(dt=None):
year = dt.year
month = dt.month
if month >= 6:
year = year + 1
return datetime(year=year,month=1,day=1)
leie_df['excl_year'] = leie_df.excldate.apply(lambda x: round_to_year(x))
In [26]:
# Make exclusion dict
# 1215053665 has 2 exclusions, so sort also df to get latest year
excl_year_dict = dict([npi, year] for npi, year in zip(leie_df.sort_values(by='excl_year').npi, leie_df.sort_values(by='excl_year').excl_year))
In [27]:
# Get label as 0 or 1
partB_df['excl_year'] = partB_df['npi'].map(excl_year_dict)
partB_df['excl_year'] = partB_df['excl_year'].fillna(datetime(year=1900,month=1,day=1)) # fill NaN, physicians without exclusion, with year 1900
partB_df['year'] = pd.to_datetime(partB_df['year'].astype(str), format='%Y', errors ='ignore')
partB_df['fraudulent'] = np.where(partB_df['year'] < partB_df['excl_year'], 1, 0) # compare year vs. exclusion year to get Fraudulent
In [28]:
print("partB_df is our combined dataset with shape: {0}".format(partB_df.shape))
4. Draw Visualizations¶
In [29]:
%matplotlib inline
import seaborn as sns
import matplotlib.pyplot as plt
import plotly.figure_factory as ff
import plotly.graph_objects as go
from plotly.subplots import make_subplots
In [30]:
# Get number and amount of fraudulent services
partB_df['fraud_line_srvc_cnt'] = partB_df['line_srvc_cnt']*partB_df['fraudulent']
partB_df['fraud_average_submitted_chrg_amt'] = partB_df['average_submitted_chrg_amt']*partB_df['fraudulent']
partB_df['fraud_average_medicare_payment_amt'] = partB_df['average_medicare_payment_amt']*partB_df['fraudulent']
In [31]:
# Aggregate by state
state_df = partB_df.groupby('nppes_provider_state').agg({
'line_srvc_cnt':[('total_services_count','sum')],
'fraud_line_srvc_cnt':[('total_fraud_services_count','sum')]
}).reset_index()
# Drop multi-index
state_df.columns = ['_'.join(col) for col in state_df.columns]
state_df.columns = ['provider_state', 'total_services_count', 'fraud_services_count']
In [32]:
# Get % fraud
state_df['fraud_services_pct'] = state_df['fraud_services_count']/state_df['total_services_count']
state_df.head()
Out[32]:
In [33]:
np.seterr(divide = 'ignore')
fig = go.Figure(data=go.Choropleth(
locations=state_df['provider_state'],
z = np.log(state_df['fraud_services_pct'].astype(float))+0.000001, #log-scale
locationmode = 'USA-states',
colorscale = 'Reds',
colorbar_title = "Logged %",
marker_line_color='white'
))
fig.update_layout(
title_text = 'Logged Percentage of Medicare Fraudulent Service by US State (2012-2016)',
geo_scope='usa',
)
fig.show()
Heat map shows a high concentration of fraudulent Medicare claims around the Northeast and South of the US, with Texas leading all states by a wide margin.
In [34]:
# Get dummy variables for gender
partB_df2 = pd.concat([partB_df, pd.get_dummies(partB_df['nppes_provider_gender'])], axis=1)
In [35]:
# Aggregate by provider type
type_df = partB_df2.groupby('provider_type').agg({
'line_srvc_cnt':['sum'],
'fraud_line_srvc_cnt':['sum'],
'M':['sum'],
'F':['sum'],
'average_submitted_chrg_amt':['median'], #since distribution skewed right
'average_medicare_payment_amt':['median'],
'fraud_average_submitted_chrg_amt':['max'],
'fraud_average_medicare_payment_amt':['max']
}).reset_index()
# Drop multi-index
type_df.columns = ['_'.join(col) for col in type_df.columns]
type_df.columns = ['provider_type', 'total_services_count', 'fraud_services_count','male_count','female_count',
'avg_submitted_chrg_amt','avg_medicare_payment_amt', 'fraud_avg_submitted_chrg_amt','fraud_avg_medicare_payment_amt']
In [36]:
# Sorting
type_df = type_df.sort_values('fraud_services_count',ascending=False)[:15] #get top 15 fraudulent types
type_df = type_df.sort_values('total_services_count',ascending=True).reset_index(drop=True) #re-sort by total services
# Add some fields
type_df['non_fraud_services_count'] = type_df['total_services_count'] - type_df['fraud_services_count']
type_df['fraud_services_pct'] = (type_df['fraud_services_count']/type_df['total_services_count'])*100
type_df.head()
Out[36]:
In [37]:
# Get 2015 data only for speed
partB_2015_df = partB_df2[partB_df2['year'] == datetime(year=2015,month=1,day=1)]
top_7_types = type_df['provider_type'][:7].tolist()
for p_type in top_7_types:
#Eliminate 0 payments then log
x = np.log(partB_2015_df[(partB_2015_df.provider_type == p_type) & (partB_2015_df.average_submitted_chrg_amt!=0)]['average_submitted_chrg_amt'])
fraud_x = np.log(partB_2015_df[(partB_2015_df.provider_type == p_type) & (partB_2015_df.fraud_average_submitted_chrg_amt!=0)]['fraud_average_submitted_chrg_amt'])
fig,ax = plt.subplots(figsize=(12,5))
sns.distplot(x,label='Non-fraudulent', hist=False, rug=False)
sns.distplot(fraud_x,label='Fraudulent', hist=False, rug=False)
ax.set(
title ='Distribution of Submitted Charge Amount - Fraud vs. Non-fraud - '+ p_type,
xlabel = 'Log of USD Payment Amount',
ylabel = 'Count'
)
plt.show()
As shown in histogram comparisons, no obvious evidence to say that the average fraudulent submitted charge is more expensive than non-fraudulent charge across the Top 15 Fraudulent categories.
This does not support the hypothesis I previously had - perhaps because doctors are pretty good at hiding wrongful claims and not making it obvious by listing unreasonably high costs.
In [38]:
for p_type in top_7_types:
#Eliminate 0 payments
x = partB_2015_df[(partB_2015_df.provider_type == p_type) & (partB_2015_df.average_medicare_payment_amt!=0)]['average_medicare_payment_amt']
fraud_x = partB_2015_df[(partB_2015_df.provider_type == p_type) & (partB_2015_df.fraud_average_medicare_payment_amt!=0)]['fraud_average_medicare_payment_amt']
fig,ax = plt.subplots(figsize=(12,5))
sns.distplot(x,label='Non-fraudulent', hist=False, rug=False)
sns.distplot(fraud_x,label='Fraudulent', hist=False, rug=False)
ax.set(
title ='Distribution of Medicare Payment Amount - Fraud vs. Non-fraud - '+ p_type,
xlabel = 'Log of USD Payment Amount',
ylabel = 'Count'
)
plt.show()
Same conclusion can be made here about Medicare Payment amount.
In [39]:
partB_2015_df = partB_df[partB_df['year'] == datetime(year=2015,month=1,day=1)]
fig, ax = plt.subplots(figsize=(15,7))
sns.heatmap(partB_2015_df.corr(method='pearson'), annot=True, fmt='.4f',
cmap=plt.get_cmap('coolwarm'), cbar=False, robust=True)
ax.set_yticklabels(ax.get_yticklabels(), rotation="horizontal")
ax.set(
title ='Correlation Heatmap in 2015 (only Continuous Variables)',
)
plt.show()
No insights are particularly interesting here, outside of the correlations we would normally expect.
In [40]:
# Get categorical variables (except location)
for col in ['nppes_provider_gender','provider_type']:
partB_2015_df = pd.concat([partB_2015_df, pd.get_dummies(partB_2015_df[col], drop_first= True)], axis=1)
partB_2015_df = partB_2015_df.drop(col, 1)
In [42]:
def get_redundant_pairs(df):
'''Get duplicate pairs to drop in correlation matrix after unstacking'''
pairs_to_drop = set()
cols = df.columns
for i in range(0, df.shape[1]):
for j in range(0, i+1):
pairs_to_drop.add((cols[i], cols[j]))
return pairs_to_drop
def get_correlations(df):
'''Get biggest correlations'''
au_corr = df.corr().unstack() #unstack
labels_to_drop = get_redundant_pairs(df)
au_corr = au_corr.drop(labels=labels_to_drop)
return au_corr
#get relevant columns
cols = partB_2015_df.columns[20:].tolist() + ['line_srvc_cnt','bene_unique_cnt','average_submitted_chrg_amt','fraudulent']
corr = get_correlations(partB_2015_df[cols])
In [43]:
corr = corr.to_frame().reset_index() #to frame
corr.columns = ['Variable A', 'Variable B', 'Correlation']
def color_df(value):
'''Color red if positive, green if negative'''
if value < 0:
color = 'red'
elif value > 0:
color = 'green'
else:
color = 'black'
return 'color: %s' % color
corr = corr.reindex(corr['Correlation'].abs().sort_values(ascending=False).index).reset_index(drop=True)
corr[:15].style.applymap(color_df, subset=['Correlation'])
Out[43]:
Again, too many interesting conclusions can be made here, apart from the fact that some medical fields/types are dominated by certain gender. The very top, Line Service Count (number of services performed) and Unique Beneficiary Count (number of distinct beneficiaries), make sense as having a positive correlation over 0.5.
Most notable is that Ambulatory Surgical Center and Anesthesiology are quite "expensive" and very positively correlated to Submitted Charge Amount -- an important insight as they are among the high flyers of physician fraud.
In [44]:
fig = go.Figure()
col_layout_dict = {'non_fraud_services_count': ['Non-fraud Service','rgba(50, 171, 96, 0.6)'],
'fraud_services_count': ['Fraud Service','rgb(255, 0, 0)']} #dict for layout
for col in ['non_fraud_services_count','fraud_services_count']:
fig.add_trace(go.Bar(
y=type_df['provider_type'],
x=type_df[col],
name=col_layout_dict[col][0],
marker=dict(
color=col_layout_dict[col][1],
),
orientation='h',
))
fig.update_layout(
barmode = 'stack',
title = 'Top 15 Fraudulent Medicare Category by Service Count',
paper_bgcolor='white',
plot_bgcolor='white',
yaxis=dict(
showgrid=False,
showline=False,
showticklabels=True,
domain=[0, 0.95],
),
xaxis=dict(
zeroline=False,
showline=False,
showticklabels=True,
showgrid=True,
domain=[0, 0.95],
),
xaxis_title_text='Count',
)
annotations = [] #annotate with %
x = type_df['fraud_services_count']+type_df['non_fraud_services_count']+100000000
y_p = np.round(type_df['fraud_services_pct'].tolist(), decimals=2)
for y_p, x, y in zip(y_p,x,type_df['provider_type']):
annotations.append(dict(xref='x1', yref='y1',
y=y, x=x,
text=str(y_p) + '%',
font=dict(family='Arial', size=12,
color='rgb(255, 0, 0)'),
showarrow=False))
annotations.append(dict(xref='paper', yref='paper',
x=-0.2, y=-0.209,
text='Combined CMS and LEIE data' +
'to label the leading Fraudulent physician categories (15 Feb 2020)',
font=dict(family='Arial', size=10, color='rgb(150,150,150)'),
showarrow=False))
fig.update_layout(annotations=annotations)
fig.show()
By our definition, Anesthesiology, General Practice, Pain Management, and Getriatric Medicine have a high percentage of fraudulent services performed by excluded physicians.
Particularly, 65% of Pain Management cases were wrongfully performed (an overinflated statistic as we qualified ALL previous services performed by an excluded physician as fraudulent if they were on the list - the reality is unlikely for ALL). Regardless, it makes sense that the high flyers would be General Practice, Pain Mangement, and Geriatric Medicine (the elderlies are the most vulnerable).
In [45]:
fig = go.Figure()
#dict for layout
col_layout_dict = {'female_count': ['Female Physicians ','#ffcdd2'],
'male_count': ['Male Physicians','#A2D5F2']}
for col in ['female_count','male_count']:
fig.add_trace(go.Bar(
y=type_df['provider_type'],
x=type_df[col],
name=col_layout_dict[col][0],
marker=dict(
color=col_layout_dict[col][1],
),
orientation='h',
))
fig.update_layout(
barmode = 'stack',
title = 'Top 15 Fraudulent Medicare Category by Service Count and Gender',
paper_bgcolor='white',
plot_bgcolor='white',
yaxis=dict(
showgrid=False,
showline=False,
showticklabels=True,
domain=[0, 0.90],
),
xaxis=dict(
zeroline=False,
showline=False,
showticklabels=True,
showgrid=True,
domain=[0, 0.90],
),
xaxis_title_text='Count',
)
fig.show()
The top fraudulent fields/categories are dominated by Male physicians.
Clinical Laboratory and Ambulance Service Provider has a lot of line services but require fewer doctors - so they did not have enough quantity to show on the bar plot?
5. Prep Data for Training¶
- Random undersample data with 50:50 class distributions (reduce the majority class to minority until 1:1)
- One hot encoding
In [46]:
partB_df.columns
Out[46]:
In [47]:
#drop variables that we are not using
partB_train_df = partB_df.drop(columns=[
'fraud_average_medicare_payment_amt', #for visual only
'fraud_average_submitted_chrg_amt', #for visual only
'fraud_line_srvc_cnt', #for visual only
'year',
'excl_year',
'hcpcs_drug_indicator',
'place_of_service',
'nppes_provider_city',
'nppes_provider_country',
'hcpcs_description',
'nppes_provider_zip'
])
In [152]:
a = partB_train_df.fraudulent.value_counts()
display(a)
print('Fraudulent physicians are: {0}% of all data'.format(str(np.round((a[1]/a[0])*100,decimals = 6))))
In [49]:
display(partB_train_df.head())
print("We have 9 features, including 4 categorical variables in \n {0}".format(partB_train_df.columns[1:5].tolist()))
In [146]:
def undersample(df, category, split=0.5):
'''Random undersampling based on a category and class split (% of dataset belonged to majority class)'''
fraud_number = df[category].sum()
non_fraud_indices = df[df[category] == 0].index #get indices of non-fraud
random_indices = np.random.choice(non_fraud_indices, size=int(round(fraud_number*(split/(1-split)), 0)),
replace=False) #sample at random without replacement
fraud_indices = df[df[category] == 1].index #get indices of fraud
under_sample_indices = np.concatenate([fraud_indices, random_indices])
return df.loc[under_sample_indices]
data = undersample(partB_train_df, 'fraudulent')
#print
a = data.fraudulent.value_counts()
display(a)
print('Fraudulent physicians are {0}% of all data.'.format(str(np.round((a[1]/(a[0]+a[1]))*100,decimals = 2))))
In [144]:
# # We are NOT normalizing the data
# from sklearn import preprocessing
# def scale_predictors(df):
# '''Takes in df and returns normalized data [0,1] '''
# x = df.values #numpy array
# min_max_scaler = preprocessing.MinMaxScaler()
# x_scaled = min_max_scaler.fit_transform(x)
# return pd.DataFrame(x_scaled, columns=df.columns, index=df.index)
In [147]:
# One-hot encoding
def one_hot_encode(df, cols):
'''Encodes categorical variables in df with cols as an array'''
for col in cols:
df = pd.concat([df, pd.get_dummies(df[col], drop_first= True)], axis=1)
return df
cols = ['provider_type', 'nppes_provider_state', 'hcpcs_code', 'nppes_provider_gender']
data = one_hot_encode(data, cols)
data = data.drop(columns=cols, axis=1).reset_index(drop=True) #drop column that's been encoded
In [148]:
display(data.shape)
print('There are {0} predictors after turning categories into dummy variables.'.format(data.shape[1]-2))
6. Model:¶
In [149]:
data.head()
Out[149]:
In [157]:
data.fraudulent.value_counts()
Out[157]:
In [159]:
import statsmodels.api as sm
cols = [col for col in data.columns if col not in ['npi', 'fraudulent']]
X = data[cols]
y = data['fraudulent']
logit_model=sm.Logit(y,X)
result=logit_model.fit(method='bfgs')
print(result.summary2())
Future Steps¶
1) Split train vs. test set - understand how well a model performs and its evaluation metrics (precision, recall, AUC).
2) Here, we only tried a 50:50 class distribution for fraud vs. non fraud. It will be worth to experiment with different class splits and compare results.
3) Try other classification models that previous literature has found to be productive (XGBoost, SVM).
4) Try a neural net.
References¶
Coding:
- https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.get_dummies.html
- https://plot.ly/python/
- https://docs.scipy.org/doc/numpy-1.14.0/reference/generated/numpy.random.choice.html
- http://blog.madhukaraphatak.com/class-imbalance-part-2/
Papers:
- Bauder & Khoshgoftaar (2018): https://www.google.com/url?sa=t&rct=j&q=&esrc=s&source=web&cd=2&ved=2ahUKEwiqkuzq--bnAhWQmeAKHXxHDPAQFjABegQIAxAB&url=https%3A%2F%2Fwww.aaai.org%2Focs%2Findex.php%2FFLAIRS%2FFLAIRS18%2Fpaper%2Fdownload%2F17617%2F16814&usg=AOvVaw2FyoQRNVI-j_xmGqktZHr_
- Johnson & Khoshgoftaar (2019): https://www.researchgate.net/publication/334555225_Medicare_fraud_detection_using_neural_networks